Information processing apparatus, information processing system, and dialogist displaying method

ABSTRACT

An image processing apparatus comprises an image display unit for displaying a desired image in accordance with an image signal, and image pickup units disposed respectively on the left and right sides of the image display unit. An image picked up from the left side and an image picked up from the right side are displayed together to be synthesized as the desired image in an image display unit of, e.g., a terminal of a conversation partner. Display in eye-to-eye matching with the conversation partner is thereby realized. An image processing apparatus, such as a portable communication terminal, capable of realizing conversation in a natural eye-to-eye matching state with the conversation partner can be provided.

TECHNICAL FIELD

The present invention relates to an information processing apparatussuch as a portable communication terminal capable of displaying an imageof a conversation partner, an information processing system, and aconversation partner display method using the portable communicationterminal, etc. More particularly, the present invention relates to aninformation processing apparatus in which when an image of aconversation partner is displayed, the eyes of the conversation partnerare matched with the eyes of a terminal user, an information processingsystem, and a conversation partner display method using the portablecommunication terminal, etc.

BACKGROUND ART

Recently, portable communication terminals, such as cellular phones,have become quite popular. Those portable communication terminals arenow equipped with not only the conversation function by speech, but alsoother various functions, e.g., transmission/reception of electronicmail, access to the Internet, and reception of a still image or a mobileimage picked up by a camera on the conversation partner side. Nowadays,in addition to the cellular phones, two-way video communication systemsconnecting two distant places to each other via a video and voicecommunication network, such as a videoconference system and avideophone, have also expanded their use.

In some of the cellular phones and the two-way video communicationsystems, an image of the conversation partner is displayed on an imagedisplay disposed near a camera for taking a picture of the user's face,etc. The user of such a cellular phone and two-way video communicationsystem usually makes conversation while looking at the image, e.g., theface of the conversation partner, displayed on the image display.

When an image pickup device, e.g., a camera, is mounted in a relativelysmall unit such as a cellular phone, the image pickup device is disposedoutside an image display unit away in the right-and-left direction orthe vertical direction. In such a case, even if the image display unithas a small size of, e.g., about 50 mm square, the eyes (line of sight)of the picked-up face image are not directed toward the image pickupdevice so long as the user looks at the image display unit where theconversation partner is displayed. As a result, that type of equipmentdisplays, on the image display unit, the face image not matched in theline of sight. In that type of equipment, therefore, the user makesconversation with the conversation partner while looking at the faceimage not matched in the line of sight, thus resulting in the problemsthat unnaturalness is not avoidable and a realistic conversationimpression is lost.

More practically, such a disadvantage can be explained as follows withreference to FIGS. 27A, 27B, 28A and 28B. FIGS. 27A, 27B, 28A and 28Bshow examples in which the direction of line of sight is not matchedwith respect to a face image on the image display unit of the terminal.Each of these examples shows an image picked up when a camera isdisposed at a position 65 mm away from the center of the image displayunit and the user looks at the center of the image display unit whilehis face is positioned at a distance of about 25 cm.

FIGS. 27A and 27B each illustratively shows an image that is picked upby a camera disposed leftward or rightward of the image display unitbeing about 50 mm square and is displayed on the image display unit.More specifically, FIG. 27A shows an image picked up by a camera that isdisposed on the left side as viewed toward the image display unit, andFIG. 27B shows an image picked up by a camera that is disposed on theright side as viewed toward the image display unit. As seen from thosedrawings, the eyes of the face in each camera image are not directedtoward the user from the image display unit, and hence the face image isunnatural in conversation.

Similarly, FIGS. 28A and 28B each illustratively shows an image that ispicked up by a camera disposed above or below the image display unitbeing about 50 mm square and is displayed on the image display unit.More specifically, FIG. 28A shows an image picked up by a camera that isdisposed on the upper side of the image display unit, and FIG. 28B showsan image picked up by a camera that is disposed on the lower side of theimage display unit. In any of these cases, the eyes of the face in eachcamera image are not directed toward the user from the image displayunit, and hence the face image is unnatural in conversation.

To overcome that unnaturalness, various eye-to-eye matching techniqueshave been proposed in the past.

According to one of those techniques, a small half mirror substantiallyin the form of a flat plate is disposed on the surface of an imagedisplay unit so that an image displayed on the image display unit passesthe small half mirror for presenting display, while an image of the useris received by an image pickup camera which captures light reflected bythe surface of the small half mirror. This technique can provideeye-to-eye matching display because the eyes of the user looking at theimage display unit are matched with an optical axis of light incidentupon the small half mirror before the light is reflected by the mirror.

According to another technique, by way of example, an image display unithaving a light transmissible structure is provided and a camera isdisposed on the backside of the image display unit relative to the user.This technique repeatedly brings the image display unit into a displaystate and a transmitted state in a time-division manner, and causes thecamera to pick up an image of the user when the image display unit is inthe transmitted state. Further, according to this technique, during aperiod other than the transmitted state, a desired video signal is sentto the image display unit and an image of the conversation partner isdisplayed. This technique can also provide eye-to-eye matching displaybecause the direction of light emerging from the image display unit ismatched with an optical axis of light passing through the image displayunit and entering the camera disposed on the backside of the imagedisplay unit.

As still another technique, there is known a display and image pickupdevice as disclosed in Japanese Unexamined Patent ApplicationPublication No. 4-167690. In this display and image pickup device, anumber of small holes are formed all over the surface of an imagedisplay unit. One end of an optical fiber is positioned to face each ofthe small holes, and the other end of each optical fiber is connected toa camera. This display and image pickup device can also provideeye-to-eye matching display because the positional relationship betweenthe ends of the optical fibers positioned to face the respective smallholes and the image display unit is not offset.

While the above-mentioned known techniques are realized by aligning theimage display unit with the optical axis of the image pickup device, itis further known, as a different type eye-to-eye matching technique, tosynthesize eyes themselves by the use of computer graphics, as describedin, e.g., “Tsuyoshi Yamaguchi et al., “Proposal for eye-to-eye matchingtechnique in videoconference”, Proceedings of Sixth Image SensingSymposium, p267-p272, 2000”.

In addition, a try to realize the eye-to-eye matching in a stereoscopicway is lately proposed (see, e.g., Japanese Unexamined PatentApplication Publication No. 10-75432).

The Japanese Unexamined Patent Application Publication No. 10-75432discloses a stereoscopic videophone in which image pickup units,constituted as cameras, and an image display unit are provided in ahousing put on a table. The image display unit is constituted by usingimage splitter type three-dimensional liquid crystal display deviceswithout spectacles, and the cameras are disposed in left-side andright-side positions of the image display unit.

Also, the Japanese Unexamined Patent Application Publication No.10-75432 mentions that images picked up by the two cameras disposed inthe left-side and right-side positions of the image display unit areselectively synthesized through fusion, whereby a front face imagecapable of providing a pseudo stereoscopic view can be obtained andusers can make conversation in an eye-to-eye matching state.

In the above-mentioned various techniques realized by aligning the imagedisplay unit with the optical axis of the image pickup device, the imagepickup device is arranged in some positional relationship relative tothe image display unit.

More specifically, the above-described technique using the small halfmirror requires the image pickup device to be positioned in thereflecting direction in which the light is reflected by the surface ofthe small half mirror. Also, the above-described technique of picking upa face image by using the light passing through the image display unitrequires the camera to be disposed on the backside of the image displayunit. To realize those techniques, therefore, the camera, etc. must beheld by an apparatus having an increased overall size, and a difficultyexists in realizing those techniques in cellular phones, for example.

Also, in the device such as described in the above-cited JapaneseUnexamined Patent Application Publication No. 4-167690 wherein opticalfibers are attached to small holes formed in an image display unit,because of a difficulty in assembly steps of forming the small holes inthe image display unit and inserting the ends of the optical fibers intothe respective small holes in one-to-one relation, the product price issignificantly increased.

Further, the technique of synthesizing the eyes of the communicationpartner by utilizing computer graphics, as described in the above-citedpaper “Proposal for eye-to-eye matching technique in videoconference”,dose not cause the above-mentioned problem attributable to the mountposition of the image pickup device. At the current technology level ofcomputer graphics, however, a synthesized image is still far from anactual one and unnaturalness in the line of sight of the communicationpartner cannot be yet completely eliminated.

Further, in the stereoscopic videophone disclosed in the above-citedJapanese Unexamined Patent Application Publication No. 10-75432, if theimage display unit has a size of about 14 inches, for example, thedistance from the left side to the right side of the image display unitin the lateral direction is about 30 cm. Therefore, when two cameras aredisposed in positions not interfering with the image display unit, arelatively large shift occurs between two picked-up images.Consequently, the videophone has the problem that when those two imagesare used, as they are, for stereoscopic image, a parallax is tooincreased to synthesize such a double image, as it is, through fusion.Even if the double image can be synthesized through fusion, theresulting display imposes a burden on the user's eyes and makes the userfeel fatigue.

The present invention has been accomplished in view of the state of theart set forth above, and its object is to provide, in relation to aportable information processing apparatus such as a portablecommunication terminal, an information processing apparatus, aninformation processing system, and a conversation partner displaymethod, which can realize conversation in a natural eye-to-eye matchingstate with a conversation partner. Another object of the presentinvention is to provide an information processing apparatus, aninformation processing system, and a conversation partner displaymethod, which can avoid the disadvantage resulting from a too largeparallax between two picked-up images and which can realize conversationin a natural eye-to-eye matching state with a conversation partner.

DISCLOSURE OF THE INVENTION

To achieve the above objects, the present invention provides an imageprocessing apparatus being portable and used for making conversationwhile presenting a picture, the image processing apparatus comprisingimage display means for displaying a desired image in accordance with animage signal; and image pickup means disposed respectively on the leftand right sides of the image display means.

With that image processing apparatus according to the present invention,since the image pickup means are disposed respectively on the left andright sides of the image display means, the image pickup means on theright side takes in an image representing a scene looking from arightward position in front of a user, and the image pickup means on theleft side takes in an image representing a scene looking from a leftwardposition in front of the user. The image picked up from the left sideand the image picked up from the right side are displayed together inimage display means of a conversation-partner side terminal, forexample, to provide display of the desired image. Thus, a person viewingthe image display means is caused to look both the images at the sametime, whereby the person visually perceives a synthesized image in astate that a shift of the line of sight in the left-and-right directionis compensated for. As a result, eye-to-eye matching display can berealized with no need of, particularly, aligning an optical axis of eachimage pickup means and an optical axis of the image display unit witheach other.

Also, to achieve the above objects, the present invention provides animage processing apparatus being portable and used for makingconversation while presenting a picture, the image processing apparatuscomprising a portable housing; image display means mounted in a surfaceof the housing and displaying a desired image in accordance with animage signal; and image pickup means disposed on the surface of thehousing respectively on the left and right sides of the image displaymeans.

With that image processing apparatus according to the present invention,the image processing apparatus has a structure incorporated in theportable housing, and the image pickup means disposed respectively onthe left and right sides are formed on a surface of the housing.Therefore, a thin structure can be realized as a whole, and the imageprocessing apparatus can be constructed of a small-sized and lightweighthousing.

Further, to achieve the above objects, the present invention provides animage processing apparatus being portable and used for makingconversation while presenting a picture, the image processing apparatuscomprising image display means including a mixed pattern of a pluralityof pixels for making display in accordance with a left-eye signal and aplurality of pixels for making display in accordance with a right-eyesignal; and image pickup means disposed respectively on the left andright sides of the image display means.

With that image processing apparatus according to the present invention,since the image pickup means are disposed respectively on the left andright sides of the image display means, the image pickup means can bemounted particularly with no need of aligning them with the position ofthe image display means, for example, with no need of an overlappedpositional relationship, whereby a small-sized and thin structure can berealized as a whole. Also, since the image display means includes themixed pattern of a plurality of pixels for making display in accordancewith a left-eye signal and a plurality of pixels for making display inaccordance with a right-eye signal, the user can continue conversationin an eye-to-eye matching state with the conversation partner withoutrequiring a special device, e.g., polarization spectacles.

In particular, each of the above-mentioned image processing apparatusesin three modes according to the present invention preferably furthercomprises image processing means for producing new images withinterpolation of a parallax based on two images picked up by the imagepickup means, wherein the two new images produced by the imageprocessing means are displayed on a display screen of the image displaymeans.

With that feature, in the image processing apparatuses according to thepresent invention, a parallax between the two images picked up by theimage pickup means is avoided from being too increased and hence fromcausing a trouble in display. As a result, stereoscopic display foreye-to-eye matching with the communication partner can be optimized, anda more natural, easier-to-see image can be provided.

Still further, to achieve the above objects, the present inventionprovides an information processing system comprising a plurality ofportable information processing terminals used for making conversationwhile presenting a picture, each of the information processing terminalscomprising image display means capable of displaying an image includinga face of a conversation partner, and image pickup means disposedrespectively on the left and right sides of the image display means, theinformation processing terminals being able to perform communicationtherebetween.

With that image processing system according to the present invention,since the image pickup means are disposed respectively on the left andright sides, the image pickup means on the right side takes in an imagerepresenting a scene looking from a rightward position in front of auser, and the image pickup means on the left side takes in an imagerepresenting a scene looking from a leftward position in front of theuser. The image picked up from the left side and the image picked upfrom the right side are displayed together in image display means of aninformation processing terminal on the conversation partner side toprovide display of the desired image. Thus, a person viewing the imagedisplay means is caused to look both the images at the same time,whereby a synthesized picture can be obtained in a state that eye-to-eyematching is held.

In particular, in the information processing system according to thepresent invention, each of the information processing terminalspreferably further comprises image processing means for producing newimages with interpolation of a parallax based on two images picked up bythe image pickup means, and the two new images produced by the imageprocessing means are displayed on the display screen of the imagedisplay means in the information processing terminal belonging to theconversation partner.

With that feature, in the information processing system according to thepresent invention, a parallax between the two images picked up by theimage pickup means is avoided from being too increased and hence fromcausing a trouble in display. As a result, stereoscopic display foreye-to-eye matching with the communication partner can be optimized, anda more natural, easier-to-see image can be provided.

Still further, to achieve the above objects, the present inventionprovides a conversation partner display method comprising an imagetaking-in step of taking in images of a user by a pair of image pickupmeans disposed respectively on the left and right sides of the imagedisplay means in a portable terminal; and a display step of displayingthe taken-in images on image display means of a terminal belonging tothe conversation partner in eye-to-eye matching between the user and theconversation partner.

With that conversation partner display method according to the presentinvention, since the user's images are taken in by the pair of imagepickup means, a signal representing a left-side image of the user and asignal representing a right-side image of the user are obtained. Thosesignals are sent to the image display means in the terminal belonging tothe conversation partner such that pixels for the left-side image andpixels for the right-side image are displayed, for example, in a mixedway. As a result, eye-to-eye matching in display can be realized betweenthe user and the conversation partner.

In particular, the conversation partner display method according to thepresent invention preferably further comprises an image processing stepof producing new images with interpolation of a parallax based on twoimages taken in by the image taking-in step, wherein the two new imagesproduced in the image processing step are displayed on the displayscreen of the image display means in the terminal belonging to theconversation partner.

With that feature, in the conversation partner display method accordingto the present invention, a parallax between the two images picked up bythe image pickup means is avoided from being too increased and hencefrom causing a trouble in display. As a result, stereoscopic display foreye-to-eye matching with the communication partner can be optimized, anda more natural, easier-to-see image can be provided.

Still further, to achieve the above objects, the present inventionprovides an image processing apparatus used for making conversationwhile presenting a picture, the image processing apparatus comprisingimage display means for displaying a desired image in accordance with animage signal; image pickup means disposed respectively on the left andright sides of the image display means; and image processing means forproducing new images with interpolation of a parallax based on twoimages picked up by the image pickup means, wherein the two new imagesproduced by the image processing means are displayed on a display screenof the image display means.

With that image processing apparatus according to the present invention,the image pickup means are disposed respectively on the left and rightsides of the image display means, new images are produced withinterpolation of a parallax based on two images picked up by the imagepickup means, and the two new images are displayed on the image displaymeans. Therefore, eye-to-eye matching display can be realized with noneed of, particularly, aligning an optical axis of each image pickupmeans and an optical axis of the image display unit with each other. Inaddition, since a parallax between the two images picked up by the imagepickup means is avoided from being too increased and hence from causinga trouble, stereoscopic display for eye-to-eye matching with thecommunication partner can be optimized, and a more natural,easier-to-see image can be provided.

Still further, to achieve the above objects, the present inventionprovides an image processing apparatus used for making conversationwhile presenting a picture, the image processing apparatus comprising ahousing; image display means mounted in a surface of the housing anddisplaying a desired image in accordance with an image signal; imagepickup means disposed on the surface of the housing respectively on theleft and right sides of the image display means; and image processingmeans for producing new images with interpolation of a parallax based ontwo images picked up by the image pickup means, wherein the two newimages produced by the image processing means are displayed on a displayscreen of the image display means.

With that image processing apparatus according to the present invention,the image processing apparatus has a structure incorporated in theportable housing, and the image pickup means disposed respectively onthe left and right sides are formed on a surface of the housing.Therefore, a thin structure can be realized as a whole, and the imageprocessing apparatus can be constructed of a small-sized and lightweighthousing.

Still further, to achieve the above objects, the present inventionprovides an image processing apparatus used for making conversationwhile presenting a picture, the image processing apparatus comprisingimage display means including a mixed pattern of a plurality of pixelsfor making display in accordance with a left-eye signal and a pluralityof pixels for making display in accordance with a right-eye signal;image pickup means disposed respectively on the left and right sides ofthe image display means; and image processing means for producing newimages with interpolation of a parallax based on two images picked up bythe image pickup means, wherein the two new images produced by the imageprocessing means are displayed on a display screen of the image displaymeans.

With that image processing apparatus according to the present invention,since the image pickup means can be mounted particularly with no need ofaligning them with the position of the image display means, for example,with no need of an overlapped positional relationship, a small-sized andthin structure can be realized as a whole. Also, the image display meansincludes the mixed pattern of a plurality of pixels for making displayin accordance with a left-eye signal and a plurality of pixels formaking display in accordance with a right-eye signal, and the imagesnewly produced with interpolation of a parallax based on the two imagespicked up by the image pickup means are displayed. Therefore,stereoscopic display for eye-to-eye matching with the communicationpartner can be optimized without requiring a special device, e.g.,polarization spectacles, and the user can continue conversation in aneye-to-eye matching state with the conversation partner while looking amore natural, easier-to-see image.

Still further, to achieve the above objects, the present inventionprovides an information processing system comprising a plurality ofportable information processing terminals used for making conversationwhile presenting a picture, each of the information processing terminalscomprising image display means capable of displaying an image includinga face of a conversation partner, and image pickup means disposedrespectively on the left and right sides of the image display means,wherein each of the information processing terminals further comprisesimage processing means for producing new images with interpolation of aparallax based on two images picked up by the image pickup means, andthe two new images produced by the image processing means are displayedon the display screen of the image display means in the informationprocessing terminal belonging to the conversation partner whencommunication is performed between the information processing terminals.

With that image processing system according to the present invention,since the image pickup means are disposed respectively on the left andright sides, the image pickup means on the right side takes in an imagerepresenting a scene looking from a rightward position in front of auser, and the image pickup means on the left side takes in an imagerepresenting a scene looking from a leftward position in front of theuser. The image picked up from the left side and the image picked upfrom the right side are displayed together in image display means of aninformation processing terminal on the conversation partner side toprovide display of the desired image. At this time, since the imagedisplay means displays the images newly produced with interpolation of aparallax based on the two images picked up by the image pickup means,stereoscopic display for eye-to-eye matching with the communicationpartner can be optimized, and a much easier-to-see, natural image can beprovided in an eye-to-eye matching state.

Still further, to achieve the above objects, the present inventionprovides a conversation partner display method comprising an imagetaking-in step of taking in images of a user by a pair of image pickupmeans disposed respectively on the left and right sides of image displaymeans in a terminal; an image processing step of producing new imageswith interpolation of a parallax based on two images taken in by theimage taking-in step; and a display step of displaying the two newimages produced in the image processing step on image display means of aterminal belonging to a conversation partner such that eye-to-eyematching in display is held between the user and the conversationpartner.

With that conversation partner display method according to the presentinvention, the user's images are taken in by the pair of image pickupmeans, and new images are produced with interpolation of a parallaxbased on the two picked-up images, whereby a signal representing aleft-side image of the user and a signal representing a right-side imageof the user are obtained. Those signals are displayed on the imagedisplay means of the terminal belonging to the conversation partner suchthat pixels for the left-side image and pixels for the right-side imageare displayed, for example, in a mixed way. As a result, stereoscopicdisplay for eye-to-eye matching with the communication partner can beoptimized, and eye-to-eye matching in display can be realized betweenthe user and the conversation partner.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view of one example of an information processingapparatus of the present invention.

FIGS. 2A to 2C are illustrations for explaining the principle of oneexample of the information processing apparatus of the presentinvention.

FIG. 3 is a block diagram showing a schematic circuit configuration ofone example of the information processing apparatus of the presentinvention.

FIG. 4 is an illustration showing a state in use of a system employingthe information processing apparatuses of the present invention.

FIG. 5 is a chart (in the x-direction) for explaining an emergence angleof a light beam from an image display unit of the information processingapparatus of the present invention.

FIG. 6 is a chart (in the y-direction) for explaining an emergence angleof a light beam from the image display unit of the informationprocessing apparatus of the present invention.

FIG. 7 shows a camera mount position around the image display unit inone example of the information processing apparatus of the presentinvention.

FIG. 8 is an illustration showing a pixel arrangement of the imagedisplay unit in one example of the information processing apparatus ofthe present invention.

FIGS. 9A to 9E show array pattern examples of left-eye pixels (L) andright-eye pixels (R) of the image display unit in one example of theinformation processing apparatus of the present invention.

FIG. 10 is an illustration showing a construction of the image displayunit at pixels and thereabout in one example of the informationprocessing apparatus of the present invention.

FIG. 11 is a sectional view of the image display unit in one example ofthe information processing apparatus of the present invention, the viewshowing the case in which there is no shift between an axis of a lightemitting device and an axis of a microlens.

FIG. 12 is a sectional view of the image display unit in one example ofthe information processing apparatus of the present invention, the viewshowing the case in which there is a shift between the axis of the lightemitting device and the axis of the microlens.

FIGS. 13A and 13B are illustrations showing spreading of a light beam bythe light emitting device and the microlens in one example of theinformation processing apparatus of the present invention, in which FIG.13A shows a simulation result of spreading of a light beam when the axisof the light emitting device and the axis of the microlens are alignedwith each other, and FIG. 13B shows a simulation result of spreading ofa light beam when the axis of the light emitting device and the axis ofthe microlens are not aligned with each other.

FIGS. 14A to 14C are graphs showing optical calculation resultsdepending on the size and positional relation of the light emittingdevice and the microlens in one example of the information processingapparatus of the present invention, in which FIG. 14A is a graph showingan optical calculation result of the relationship between a diameter ofthe light emitting device and a light spreading angle, FIG. 14B is agraph showing a calculation result of the relationship between a lensposition in the direction of an optical axis and spreading of a lightbeam, and FIG. 14C is a graph showing a calculation result of therelationship between a position of the light emitting device in thein-plane direction and a light exit angle.

FIGS. 15A and 15B show an example of the microlenses used in one exampleof the information processing apparatus of the present invention, inwhich FIG. 15A is a front view and FIG. 15B is a sectional view.

FIG. 16 is a conceptual view for explaining the concept of imageprocessing as the function in one example of the information processingapparatus of the present invention, the view being referenced to explainimage processing to find out an area of an image closest to a template.

FIG. 17 is an illustration indicating parameters for a parallax.

FIGS. 18A and 18B show the image display unit in another example of theinformation processing apparatus of the present invention, in which FIG.18A is a sectional view a portion near a light emitting device and acone-shaped microlens, and FIG. 18B is a perspective view of thecone-shaped microlens.

FIG. 19 is a chart showing parameters referenced to explain therelationship between emitted light and an exit angle when the microlens,etc. are employed.

FIG. 20 is a graph for explaining the relationship between a ratio of apartial spherical area to a total spherical area and an exit angle φwhen the microlens, etc. are employed.

FIG. 21 is a sectional view showing the image display unit in stillanother example of the information processing apparatus of the presentinvention.

FIGS. 22A and 22B are illustrations showing schemes of experiments toexamine the relationship between the direction of line of sight and animpression of an eye-to-eye matching, in which FIG. 22A shows the casein which a stereoscopic image is provided, and FIG. 22B shows the casein which a monaural image is provided.

FIGS. 23A and 23B are graphs showing results of the experiments shown inFIGS. 22A and 22B, in which FIG. 23A shows the case in which astereoscopic image is provided, and FIG. 23B shows the case in which amonaural image is provided.

FIG. 24 is a conceptual view for explaining the concept of interpolatinga parallax in the image processing apparatus of the present invention,the view being referenced to explain a process for producing, based ontwo images picked up by two cameras disposed on both left and rightsides of the image display area, a new image that has a smaller parallaxas if it is picked up by two virtual cameras disposed at a smallerspacing than that between the two actual cameras.

FIG. 25 is a conceptual view for explaining the concept of imageprocessing as the function in one example of an information processingapparatus of the present invention, the view being referenced to explainimage processing to obtain corresponding points in two images picked upby two cameras disposed on both left and right sides of the imagedisplay area.

FIG. 26 is an illustration showing one example of an image produced bymoving an original image by the number of pixels corresponding to ½ of ashift representing a parallax, the view being referenced to explain aprocess of producing an image comparable to that picked up from aposition at the middle of the two cameras used for picking up the twoimages.

FIGS. 27A and 27B are each an illustration showing a display unit imagewhen an object image is picked up by a camera disposed one of the leftand right sides of a screen having a substantially predetermined squaresize, in which FIG. 27A shows the case in which the camera is disposedon the left side, and FIG. 27B shows the case in which the camera isdisposed on the right side.

FIGS. 28A and 28B are each an illustration showing a display unit imagewhen an object image is picked up by a camera disposed one of the upperand left sides of a screen having a substantially predetermined squaresize, in which FIG. 28A shows the case in which the camera is disposedon the upper side, and FIG. 28B shows the case in which the camera isdisposed on the lower side.

BEST MODE FOR CARRYING OUT THE INVENTION

A practical embodiment implementing the present invention will bedescribed in detail below with reference to the drawings.

This embodiment relates to a portable communication terminal capable ofdisplaying a face image of a conversation partner. As shown in FIG. 1, aportable communication terminal 10 serving as an information processingapparatus has a housing 12 with a size allowing a user to hold it by onehand, and enables the user to make conversation while holding thehousing 12 with the lengthwise direction of the housing 12 orientedvertically. An image display unit 11 with a size in the range of about10 mm square to 100 mm square is disposed on the front side of thehousing 12 such that its display screen is exposed to the outside. Aleft camera 13L and a right camera 13R each serving as an image pickupunit are disposed respectively on the left and right sides of the imagedisplay unit 11. Further, a minor monitor 14 for monitoring an imagecondition of a user's face, etc. is disposed below the image displayunit 11, and an indicator 17 is disposed laterally of the minor monitor14. When the user's face is deviated far away from an image pickup area,the indicator 17 is blinked, for example, to indicate that fact. Abutton 15 and input keys 16 are arranged below the minor monitor 14. Inthe portable communication terminal 10, with operation of the button 15and the input keys 16, the user can, for example, dial the phone number,enter characters for electronic mail, and control the functions of theportable communication terminal 10.

The left camera 13L and the right camera 13R each serving as an imagepickup unit are disposed respectively on the left and right sides of theimage display unit 11. The left camera 13L is disposed to pick up apicture from a position in front of the user holding the portablecommunication terminal 10 slightly offset to the left, and the rightcamera 13R is disposed to pick up a picture from a position in front ofthe user slightly offset to the right. The left camera 13L and the rightcamera 13R are each constituted as an optical system in a combination ofa MOS (Metal-Oxide Semiconductor) image pickup device or a CCD (ChargeCoupled Device) image pickup device with a lens, etc. Practically, inthe portable communication terminal 10, a small-sized and lightweightsolid-state image pickup device is used, by way of example, as each ofthe left camera 13L and the right camera 13R.

The image display unit 11 is constituted as including a mixed pattern ofa plurality of pixels for making display in accordance with a left-eyesignal and a plurality of pixels for making display in accordance with aright-eye signal. Particularly, the image display unit 11 includesoutput means for outputting lights to both eyes of the userindependently of each other. In this embodiment, the output meanscomprises a light emitting portion for emitting desired light inaccordance with the left-eye signal or the right-eye signal, and an exitangle control portion for controlling light from the light emittingportion to exit in the direction of a predetermined angle. The lightemitting portion is constituted by a plurality of light emitting diodesin an array, and the exit angle control portion is constituted as amicrolens array formed by arraying a plurality of small lenses in amatrix pattern. The above-described structure of the output means ismerely one example, and the structure of the light emitting portion maybe constituted, as other examples, by using liquid crystal displaydevices, organic EL (Electronic Luminescent) devices, plasma displaydevices, etc., or by employing a transmitted or reflecting structure.The exit angle control portion can also be modified to any suitable oneof various structures as described later. A method of forming the mixedpattern of a plurality of pixels for making display in accordance withthe left-eye signal and a plurality of pixels for making display inaccordance with the right-eye signal will be described later.

A technique for sending respective signals with respect to both the eyesto one display unit for eye-to-eye matching will be first described.Generally, a detection eye and an allowance eye with respect tomisalignment of the line of sight have been studied in the field ofvideophones or the likes as discussed in, e.g., “Sato et al., No. 1998,“Studies on position of image pickup tube in videophone”, 1972 of JointMeeting of Four Electrical Societies in Japan, p2316 and “Minami, “4.2Videophone”, Journal of the Institute of Electronics and CommunicationEngineers in Japan, 11/'73, Vol. 56, No. 11, p1485-p1490”. Thosereferences show discussions and experimental results regarding alignmentof the line of sight in videophones, and mention that “a detection eyecapable of detecting an unusual feeling in the line of sight as thehuman sense is in a very narrow range of about 2° to 3° from the center,and there is no appreciable difference between the horizontal directionand the vertical direction”. Further, those references mention that “anallowance eye providing a limit of the allowable range in videophonesvaries such that a limit angle is about 4.5° in the horizontaldirection, 12° in the exactly upward vertical direction, and 8° in theexactly downward vertical direction, and that since the allowable rangeis narrow particularly in the horizontal direction, the human being issensitive to the line of sight in the left-and-right direction”.

The portable communication terminal 10 shown as the embodiment of thepresent invention is an apparatus capable of compensating for a shift inthe line of sight in the left-and-right direction that is relativelyeasily sensible, and realizing communication with satisfactory realism.FIGS. 2A to 2C are illustrations for simply showing the concept ofrealizing such an apparatus. FIG. 2A shows an image viewed from the leftcamera 13L. Because the left camera 13L is positioned on the left sideoffset from the center, the line of sight of the user appears offset tothe right. Also, FIG. 2B shows an image viewed from the right camera13R. Because the right camera 13R is positioned on the right side offsetfrom the center, the line of sight of the user appears offset to theleft. Further, FIG. 2C shows a display unit resulting when an image isdisplayed by using the image display unit 11 which includes the mixedpattern of a plurality of pixels for making display in accordance withthe left-eye signal and a plurality of pixels for making display inaccordance with the right-eye signal and which has a mechanism foroutputting lights to both the eyes of the user independently of eachother. Stated another way, the portable communication terminal 10outputs light toward the left eye of the user from each pixel of theimage display unit 11 to which the left-eye signal is dedicatedly sent,and the output light reaches the left eye of the user. Similarly, theportable communication terminal 10 also outputs light toward the righteye of the user from each pixel of the image display unit 11 to whichthe right-eye signal is dedicatedly sent, and the output light reachesthe right eye of the user. As a result, the user looks different imagesby both the eyes. However, the function of looking those images as asynthesized one works in the cerebrum of the user, and hence the usercan see the images as if an eye-to-eye matching image is displayed onthe screen, as shown in FIG. 2C, so that communication can be realizedin a natural eye-to-eye matching state. The image may be a still pictureor a mobile picture. When the portable communication terminal 10 has anallowance in band for transmission/reception, a real-time mobile picturecan also be transmitted and received. Additionally, compressed orthinned-out image data may be transmitted and received.

FIG. 3 is a block diagram showing an example of circuit configuration ofthe portable communication terminal 10. A control circuit 21 includes aCPU (Central Processing Circuit), a video memory, and an imageinformation processing circuit having required specifications. Asdescribed later, the control circuit 21 also executes signal processingto prevent an effect of hand wobbling. Data processed by the controlcircuit 21 can be transmitted via a transmitting/receiving unit 28, anddata received by the transmitting/receiving unit 28 is sent to thecontrol circuit 21. A key input unit 27 has, for example, ten keys andfunction setting buttons arranged on it. In addition, a jog dial, aninput pad, another add-on functional device, etc. may also be arrangedon the key input unit 27.

A signal from the left camera 24L and a signal from the right camera 24Rare independently supplied to the control circuit 21. In an ordinarystate in use, the signal from the left camera 24L contains image datapicked up from a position in front of the user slightly offset to theleft, and the signal from the right camera 24R contains image datapicked up from a position in front of the user slightly offset to theright. In the portable communication terminal 10, the signals from theleft camera 24L and the right camera 24R are transmitted to thetransmitting/receiving unit 28 via the control circuit 21. Those signalsare transmitted from the transmitting/receiving unit 28 and received byanother portable communication terminal. The portable communicationterminal 10 enables the user to make conversation through theabove-described transmission and reception of data. Particularly, withthis embodiment, the user can look the received data as if an eye-to-eyematching image is displayed on the image display unit 11, and hencecommunication is realized in a natural eye-to-eye matching state.

The signals from the left camera 24L and the right camera 24R are notonly sent to another portable communication terminal, but also used todisplay an image on a small monitor screen via the control circuit 21.More specifically, the signals from the left camera 24L and the rightcamera 24R are also sent to a minor monitor 25 via the control circuit21 and a driver 26. As a result, the user can look his or her face on arelatively small screen, e.g., the minor monitor 14 shown in FIG. 1, ofthe portable communication terminal 10 while making conversation. Whenthe user makes conversation without confirming his or her face on therelatively small screen, e.g., the minor monitor 14, there is apossibility depending on an angle of the portable communication terminal10 that the user's face cannot be captured at the center of any of theleft camera 24L and the right camera 24R, or that the user's face isdeviated from the image pickup range of the camera. By makingconversation while confirming the image on the minor monitor 25, theuser of the portable communication terminal 10 can continue theconversation in a natural eye-to-eye matching state with theconversation partner. Thus, the minor monitor 25 is a device forprompting visual confirmation. In the portable communication terminal10, since the signals from the left camera 24L and the right camera 24Rare subjected to steady image processing, such as hand wobblingpreventive processing, in the control circuit 21, the indicator 29 maybe blinked when the user's face is not properly displayed.Alternatively, control may be executed to display that condition on thescreen of the minor monitor 25 or a main monitor 22.

A signal from the conversation partner is sent to the control circuit 21from the transmitting/receiving unit 28 and is output from the controlcircuit 21 after being separated into a signal LE for the left eye and asignal RE for the right eye. The left-eye signal LE is sent to aleft-eye driver 23L for driving pixels for the left eye, and theright-eye signal RE is sent to a right-eye driver 23R for driving pixelsfor the right eye. The main monitor 22 for displaying the face of theconversation partner is driven by both the signals from the left-eyedriver 23L and the right-eye driver 23R. On the main monitor 22, asdescribed above, the left-eye pixels and the right-eye pixels arearrayed in a mixed pattern, for example, such that pixels in even linesare for the right eyes and pixels in odd lines are for the left eye, andtherefore different images are independently displayed per group ofpixels. On the side looking the main monitor 22, however, the functionof looking the different images as a synthesized one works in thecerebrum of the user, and hence the user can see the images as if aneye-to-eye matching image is displayed on the screen, as shown in FIG.2C, so that communication can be realized in a natural eye-to-eyematching state.

Note that the block diagram of FIG. 3 merely shows one example of theportable communication terminal 10 and may have any other suitablecircuit configuration. For example, in the portable communicationterminal 10, only one of the left-eye signal LE and the right-eye signalRE may be sent to both the drivers 23L, 23R with, e.g., buttonoperation. Alternatively, the signal for the main monitor 22 and thesignal for the minor monitor 25 may be controlled to change over with,e.g., button operation so that the user's face is displayed on the mainmonitor 22 in an eye-to-eye matching state. Also, the portablecommunication terminal 10 may be modified so as to produce voices in astereoscopic manner in addition to the image. Furthermore, in theportable communication terminal 10, the left-eye signal and theright-eye signal may be reproduced only when an occasion requires, andan image may be displayed with a signal not dedicated for the left eyeor the right eye in the ordinary case.

FIG. 4 is an illustration showing a situation in which two persons aretalking through two portable communication terminals 35, 38 each havingthe same construction as the portable communication terminal 10described above. Assuming, in FIG. 4, that a user 33 and a conversationpartner 34 are communicating with each other, the portable communicationterminal 35 held by the user 33 is equipped with an image display unit36 and a pair of cameras 37L, 37R. The face of the conversation partner34 is displayed on the image display unit 36, and a displayed face imageis in a natural eye-to-eye matching state based on the principledescribed above with reference to FIGS. 2A to 2C. On the other hand, theportable communication terminal 38 held by the conversation partner 34is equipped with an image display unit 39 and a pair of cameras 40L,40R. The face of the user 33 is displayed on the image display unit 38,and a displayed face image is similarly in a natural eye-to-eye matchingstate based on the principle described above with reference to FIGS. 2Ato 2C.

With reference to FIGS. 5 and 6, a description is now made of astructure for controlling light emitted from the light emitting portionof the image display unit in the portable communication terminal of thisembodiment to exit in the direction of a predetermined angle.

FIG. 5 is a chart looking, from above, the image display unit of theportable communication terminal of this embodiment.

In FIG. 5, an area b1 represents a range where light radiated from orreflected by a pixel at a left end of a display screen of the imagedisplay unit is sufficiently strong for permitting the user to clearlyview it. Likewise, an area b2 represents a range where light radiatedfrom or reflected by a pixel at a right end of the display screen of theimage display unit is sufficiently strong. Between those two pixels, anangle θ is continuously changed such that corresponding light ranges arealmost overlapped with each other in a plane at a distance L₁ away fromthe display screen of the image display unit. The distance L₁ is adimension assumed to be a distance at which the user looks the displayscreen in the ordinary case. Herein, the distance L₁ is supposed to be250 mm, i.e., the so-called distance of distinct vision for the humanbeing. Also, a distance L₂ is a dimension assumed in consideration ofthe case that the user looks the display screen while stretching thearm. Herein, the distance L₂ is supposed to be 400 mm. Black points b3,b4 represent respective positions of a left eye E_(L1) and a right eyeE_(R1) of the user at the distance L₁ who looks the display screen ofthe portable communication terminal. Further, black points b3′, b4′represent respective positions of both the eyes at the distance L₂.

As is apparent from FIG. 5, the lights in the areas b1, b2 do not enterthe right eye, and can be seen only by the left eye. Likewise, pixelscapable of been seen only by the right eye can be set by reversing thelight angle in the left-and-right direction. Accordingly, a stereoscopicview can be obtained by displaying respective images for the left eyeand the right eye per line or pixel with respect to all the displaypixels.

A distance D is a dimension of the display screen of the image displayunit in the horizontal direction. In general, the image display unit ofportable equipment has a width from about 20 mm to 80 mm to meet ademand for a hand-held unit. In this example, the distance D is supposedto be 40 mm. Also, θ₁(=−9.57±5°) and θ₂(=0.43±5°) are design referencevalues in this example and are set on an assumption of light spreadingbeing 10° so that light sufficiently reaches the eye, while ensuringthat the lights for both the eyes are neither mixed nor spaced away fromeach other as far as possible at the middle between both the eyes at aposition of L₁=250 mm. Those design values also allow light tosufficiently reach the eye even at the distance L₂ of 400 mm. Thedistance at which a part of the display screen disappears theoreticallyunder that setting is 202 mm at minimum and 657 mm at maximum based onthe geometry. The light spreading angle may be larger than 10° so longas the lights reaching both the eyes are separable from each other.However, a larger spreading angle requires a larger light exit angle andraises a difficulty in optical design. Further, because the equipment asan application of the present invention is assumed to be a personaluser, not spreading the light more than necessary is advantageous fromthe viewpoints of privacy protection and a reduction of energyconsumption.

Regarding the vertical direction, as shown in FIG. 6, control issimilarly performed so that light reaches the eye position. In thevertical direction, an angle φ is used as an angle parameter for theexit direction. Since it is assumed that there is no positionaldifference between both the eyes, the black points b3, b4 are located atthe same position (L₁) and the black points b3′, b4′ are located at thesame position (L₂).

While such a design process enables the image display unit to be capableof displaying an image in an eye-to-eye matching state, it can befurther generalized using formulae given below. First, a distance E_(R1)corresponding to half of the spacing between both the eyes is comparedwith a distance Dx_(max) from the center to the end of the displayscreen of the image display unit. In other words, a comparison is madeas expressed by the following formulae (1):if Dx_(max)<E_(R1)if Dx_(max)>E_(R1)   (1)

If the upper condition of the formulae (1) is satisfied, this indicatesthe case that the distance Dx_(max) from the center to the end of thedisplay screen of the image display unit is shorter than the distanceE_(R1) corresponding to half of the spacing between both the eyes, i.e.,the case that the display screen of the image display unit has a smallsize. In this case, the light exit angles are set as expressed by thefollowing formulae (2):θ_(L max)=−tan⁻¹(Dx/L ₁)θ_(L min)=θ_(max)−10θ_(Rmax), θ_(Rmin) are symmetrical about center (yz-plane)φ_(mid)=(−tan⁻¹(Dy/L ₁)−tan⁻¹(Dx/L ₂))/2φ_(max)=φ_(mid)+5φ_(max)=φ_(mid)−5   (2)

In the formulae (2), the distance L₁ is a dimension assumed to be adistance at which the user looks the display screen in the ordinarycase, e.g., the distance of distinct vision for the human being. Also,the distance L₂ is a dimension assumed in consideration of the case thatthe user looks the display screen while stretching the arm. Further, thedistance Dx is a distance in the x-direction (horizontal direction), andthe distance Dy is a distance in the y-direction (vertical direction).Stated another way, calculation can be made on an assumption that apixel for which the direction of the exit angle is to be determined ispositioned at (Dx, Dy).

On the other hand, if the lower condition of the formulae (1) issatisfied, this indicates the case that the distance Dx_(max) from thecenter to the end of the display screen of the image display unit islarger than the distance E_(R1) corresponding to half of the spacingbetween both the eyes, i.e., the case that the display screen of theimage display unit has a large size. In this case, the light exit anglesare set as expressed by the following formulae (3):θ_(L max)=−tan⁻¹((Dx−E _(R2))/L ₁)θ_(L min)=−tan⁻¹((Dx−E _(L1))/L ₂)θ_(Rmax), θ_(Rmin) are symmetrical about center (yz-plane)φ_(mid)=(−tan⁻¹(Dy/L ₁)−tan⁻¹(Dx/L ₂))/2φ_(max)=φ_(mid)+5φ_(max)=φ_(mid)−5   (3)

In the formulae (3), each parameter has the same setting as that in theformulae (2).

By using the formulae (1) to (3), it is possible to determine thedirection of the light exit angle at an arbitrary position (Dx, Dy) onthe image display unit. If the relationship in correspondence betweenthe pixels and the right eye or the left eye is at random, the exitdirection can be determined for each pixel by calculation using theformulae (1) to (3). As another example, if the right-eye pixels and theleft-eye pixels are alternately arrayed per line, the exit direction maybe set such that several points on each line are extracted and a similarcalculation is executed for each of the points by using the formulae (1)to (3), while other points than the extracted ones are each set by alinear interpolation or another suitable process based on data of theexit angle for the extracted points.

Camera positions in the portable communication terminal of thisembodiment will be described below with reference to FIG. 7.

An image display unit 31 is of a structure comprising many pixelsarrayed in a matrix pattern and has a substantially rectangular outercontour. Cameras are disposed at least one on each of the left and rightsides. As another example, a larger number of cameras may be disposed,or different types of cameras may be disposed. In the latter case, acombination may be such that one of the left and right cameras is anordinary camera, and the other camera is a relatively simplified one forthe purpose of just synthesizing the line of sight.

Each of the cameras can be disposed in one of areas 32L, 32R shown inFIG. 7, by way of example, including not only positions horizontallyaside from left and right side ends of the image display unit 31 havinga substantially rectangular shape, but also positions within apredetermined range away from upper and lower ends of the left and rightside ends. More specifically, the areas 32L, 32R shown in FIG. 7 includenot only band-like zones horizontally extending from the left and rightside ends of the image display unit 31 and having a width H₂, but alsozones each extending vertically from one of upper and lower ends ofthose band-like zones and defined by a circle with a radius r, forexample, about the corresponding corner of the rectangular image displayunit 31. Thus, in the direction of height, each of the areas 32L, 32Rincludes a zone extending upward from the rectangular image display unit31 by a distance H₁. The reason why there is a relatively large degreein freedom for the camera position in the vertical direction is that theabove-mentioned allowance eye has a not so severe limit in the verticaldirection. In the case of the portable communication terminal that has asmall space other than the display unit, the camera position is notalways limited to a position exactly laterally of the image display unit31. The above description is similarly applied to the lower side.

The width H₂ is not limited to a particular value. For example, when theradius r is set to about 20 mm, the width H₂ can also be set to about 20mm. The above-described camera position is desirably fixed relative tothe image display unit 31 because each camera is equipped with anoptical system (not shown). Alternatively, the structure may be modifiedsuch that the camera is able to project and retract from the side of theimage display unit 31, or that both or one of a pair of cameras ismounted in place when an image is picked up. A lens made of glass orplastic is attached to a fore end of the camera. To prevent the lensfrom being damaged, for example, the camera may be covered with a coverwhen not used.

Examples of the structure of the image display unit will be nextdescribed with reference to FIGS. 8 to 12.

FIG. 8 shows a display pixel in an enlarged scale. In FIG. 8, an area 51indicated by a substantially square block corresponds to one pixel. Inthe area 51 of each pixel, four light emitting devices 52R, 52G, 52B and52G are disposed so as to position at four points in one surface of adie, respectively, which represents a number 4. The light emittingdevices 52R, 52G, 52B and 52G are each formed of a semiconductor lightemitting device, such as a light emitting diode. The light emittingdevice 52R is a device emitting red light, the light emitting device 52Gis a device emitting green light, and the light emitting device 52B is adevice emitting blue light. The green light emitting device 52G has sucha nature that it is more easily resolved by human eyes than the lightemitting devices emitting lights in other colors. Therefore, a moreuniform impression can be provided by more densely arraying the greenlight emitting devices 52G. The image display unit may be constituted,instead of those light emitting devices, by using transmission displaydevices with color filters, such as colored liquid crystals, or othersuitable reflecting display devices.

The image display unit capable of providing a required stereoscopic viewby outputting respective light beams to the left and right eyes with theabove-described pixel area 51 can be constituted by setting those pixelareas so as to produce two different images with distribution of thelight beams per line or pixel.

FIGS. 9A to 9E show examples of the pattern for distributing left andright images to realize a stereoscopic view. In each of these drawings,“L” represents a pixel for emitting light in accordance with data forthe left eye such that the light is output toward the left eye. On theother hand, “R” represents a pixel for emitting light in accordance withdata for the right eye such that the light is output toward the righteye. Note that each drawing shows only an extracted portion of fourhorizontal pixels x four vertical pixels. FIG. 9A shows a pattern inwhich the left-eye pixels represented by “L” and the right-eye pixelsrepresented by “R” are alternately arranged per horizontal line. FIG. 9Bshows a pattern in which the left-eye pixels represented by “L” and theright-eye pixels represented by “R” are alternately arranged pervertical line. FIG. 9C shows a pattern in which the left-eye pixels andthe right-eye pixels alternately appear checkerwise. In this pattern,the left-eye pixel and the right-eye pixel alternately appear per pixelin each horizontal line, and a similar pattern appears with a shift ofone pixel in the horizontal direction between one current horizontalline and the next horizontal line. FIG. 9D shows the case in which acheckered pattern is formed as in FIG. 9C, but the checkered pattern isformed in unit size of two horizontal pixels x two vertical pixels.Further, FIG. 9E shows a pattern in which the left-eye pixels and theright-eye pixels alternately appear per two pixels in each horizontalline, and a similar pattern appears with a shift of one pixel in thehorizontal direction between one current horizontal line and the nexthorizontal line. Note that the patterns of FIGS. 9A to 9E are shownmerely by way of example and may be replaced with other suitablepatterns. Also, while the overall display screen can be formed in thesame pattern, it is possible, for example, to form different patterns incentral and peripheral areas of the image display unit, or to arrange apattern for distributing lights into left and right images primarily inan area that is assumed to correspond to the position of a face image.Further, instead of distributing the left-eye pixels and the right-eyepixels in a regular pattern, the left-eye pixels and the right-eyepixels may be distributed in an irregular pattern although wiring iscomplicated.

An example using small lenses, i.e., microlenses, arranged on the lightemitting devices in one-to-one relation will be described below as oneexample of the structure of the image display unit.

FIG. 10 shows an arrangement in which four light emitting devices 63G,63B and 63R each formed of a semiconductor light emitting device, suchas a light emitting diode or a semiconductor laser, are disposed in eachpixel 61. The light emitting device 63R is a device emitting red light,the light emitting device 63G is a device emitting green light, and thelight emitting device 63B is a device emitting blue light. As mentionedabove, the green light emitting device 63G has such a nature that it ismore easily resolved by human eyes than the light emitting devicesemitting lights in other colors. Therefore, a more uniform impressioncan be provided by more densely arraying the green light emittingdevices 52G.

Small microlenses 62 each made of a spherical transparent body arearranged on the surface side of the image display unit, whichcorresponds to the light exit side of the light emitting devices 63G,63B and 63R. The microlenses 62 serve as exit angle control portions foroutputting lights from the light emitting devices 63G, 63B and 63R inthe direction of a predetermined angle toward the left eye or the righteye, and they are formed of a transparent synthetic resin such as PMMA(polymethacrylic methyl), glass, or the like. The shape of eachmicrolens is not limited to a sphere, but it may be conical, pyramidal,or rectangular. Also, the microlenses can be assembled into the imagedisplay unit by bonding a holding plate onto which the microlenses areintegrally mounted in a matrix pattern, or by positioning themicrolenses one by one. In order that each microlens 62 has the functionof controlling the light exit angle, an opening for each light emittingdevice may be formed in a shield plate or the like to be directed towardone of the user's eyes, or the position of the microlens 62 may beshifted from an optical axis of the light from corresponding one of thelight emitting devices 63G, 63B and 63R.

FIGS. 11 and 12 are each a schematic sectional view for explaining anexample of controlling the light exit direction based on the position ofthe microlens 62. FIG. 11 shows an example in which an axis of the lightemitting device 63G and an axis of the microlens 62 are aligned witheach other, and FIG. 12 shows an example in which there is a shiftbetween the axis of the light emitting device 63G and the axis of themicrolens 62.

FIG. 11 typically shows the light emitting device 63G emitting greenlight, and the light emitting device 63G has a size φ_(OPT) of about 30μm. In FIG. 11, the z-axis direction represents the direction normal tothe display screen of the image display unit, and it is herein definedas the light exit direction. The light emitting device 63G is, e.g., aGaN-based light emitting diode, and the light emitting diode emittingblue light can also be constituted by, e.g., a GaN-based semiconductor.The light emitting device 63R emitting red light may be constituted by,e.g., a GaAs-based compound semiconductor. The light emitting devices63G are each bonded onto a support substrate 65 and are arrayed in amatrix pattern at a pitch in the range of about 300 μm to 600 μm.

On the support substrate 65, a molded holding member 66 is disposedwhich serves not only as a member for holding the microlenses 62, butalso as a shield plate for limiting the direction of light from thelight emitting device to a proper angle. The molded holding member 66has openings formed corresponding to respective positions of the lightemitting device. Each of the openings has a diameter graduallyincreasing substantially in the form of a truncated cone, and themicrolens 62 is fitted to an end of the opening opposed to the lightemitting device. The molded holding member 66 and the microlenses 62 arefixedly bonded to each other, and so are the molded holding member 66and the support substrate 65. The microlenses 62 are interconnected andheld in place by holding portions 64 which hold the microlenses at theirmaximum diameter regions. A diameter φ_(LENS) of each microlens 62 isset herein to about 300 μm. With such an arrangement, a gap of adistance d is formed between the light emitting device 63G and thebottom of the microlens 62 mounted to the opening of the molded holdingmember 66, and light is introduced to the microlens 62 after passingthrough the gap.

FIG. 12 shows, as described above, an example in which there is a shiftbetween the axis of the light emitting device 63G and the axis of themicrolens 62. In FIG. 12, the light emitting device 63G is disposed in aposition shifted by a distance Δy from a line passing the center of themicrolens 62 and being parallel to the z-axis. When the light emittingdevice 63G is disposed in such a position shifted by the distance Δy,the light exiting from the light emitting device 63G is bent due to theshift from the axis of the microlens 62. Thus, the setting of thatpositional relationship enables lights to exit in respective directionstoward the right eye and the left eye. The axis of the light emittingdevice 63G and the axis of the microlens 62 can be shifted from eachother, for example, by a method of shifting the position of the lightemitting device 63 on the support substrate 65, a method of shifting theposition of the microlens 62 on it, or in the structure in which themicrolenses 62 are mounted to respective openings in the molded holdingmember 66 one by one, by a method of shifting the position of eachopening in the molded holding member 66 and hence shifting the positionof each microlens 62. The example shown in FIG. 12 employs the method ofshifting the position of the microlens 62. More specifically, themicrolenses 62 are fixed to the molded holding member 66 such that thecenter of the opening in the molded holding member 66 is not alignedwith the axis of the microlens 62. While FIG. 12 shows the shift in they-direction, a similar shift may be produced, in addition to they-direction, in any of the z-direction and the x-direction so thatlights can be output in respective directions toward the right eye andthe left eye.

FIG. 13A shows a simulation result of spreading of a light beam when theaxis of the light emitting device 63G and the axis of the microlens 62are aligned with each other, and FIG. 13B shows a simulation result ofspreading of a light beam when the axis of the light emitting device 63Gand the axis of the microlens 62 are shifted from each other. In otherwords, FIG. 13A shows the relationship between the light emitting deviceand the microlens in the structure of FIG. 11, and FIG. 13B shows therelationship between the light emitting device and the microlens in thestructure of FIG. 12. When the axis of the light emitting device 63G andthe axis of the microlens 62 are aligned with each other, as shown inFIG. 13A, the light beam spreads about the z-axis, i.e., the directionnormal to the display screen of the image display unit. However, whenthe axis of the light emitting device 63G and the axis of the microlens62 are shifted from each other, as shown in FIG. 13B, an angle is givento the light exit direction and the light beam is radiated slightlyobliquely upward as viewed in the drawing. This calculation example ison an assumption that the material of the microlens is PMMA, thediameter of the microlens is 300 μm, the diameter of the light emittingdevice is 30 μm, and the distance between the light emitting device andthe microlens is 50 μm. FIG. 13A shows the case of Δy=0, and FIG. 13Bshows the case of Δy=−15 μm. When PMMA is used as the material of themicrolens, its refractive index varies depending on wavelength. Therelationship between the refractive index and wavelength is shown in atable given below. The calculation is executed using data shown in thetable. Refractive Index of PMMA Color Blue Green Red Light F e CWavelength 486.13 546.07 656.27 (nm) Refractive  1.49776  1.49776 1.48920 Index

Furthermore, as plotted in FIGS. 14A to 14C, optical calculations areexecuted regarding the size and positional relation of the microlens andthe light emitting device.

First, a graph of FIG. 14A shows an optical calculation result of therelationship between the diameter of the light emitting device and thelight spreading angle. Conditions are similar to those in the case ofFIG. 13A. Namely, the diameter of the light emitting device is changedon condition that the material of the microlens is PMMA, the diameter ofthe microlens is 300 μm, and the distance between the light emittingdevice and the microlens is 50 μm. As seen from the result of FIG. 14A,to obtain the light spreading angle of 10° as the above-mentioned targetvalue under those conditions, the diameter of the light emitting deviceis preferably about 30 μm. Because the size of the microlens and thesize of the light emitting device are relatively compared with eachother, a size ration of the microlens to the light emitting device ispreferably set, for example, to the range of 30:1 to 5:1.

A graph of FIG. 14B shows a result of calculating the effect of thedistance between the light emitting device and the microlens upon thelight spreading angle. Conditions are similar to those in the case ofFIG. 14A. Namely, the distance d between the light emitting device andthe microlens is variable on condition that the material of themicrolens is PMMA, the diameter of the microlens is 300 μm, and thediameter of the light emitting device is 30 μm. As seen from the resultof FIG. 14B, to obtain the light spreading angle of 10°, the distancebetween the light emitting device and the microlens is preferably set toabout 50 μm.

A graph of FIG. 14C shows a calculation result of the relationshipbetween the distance Δy (=Δx), i.e., a shift between the light emittingdevice and the microlens in the lateral direction, and the light exitangle. Upper and lower limit angles represent the range of θ for thelight spreading area shown in FIG. 5. “Center” represents a valueobtained by plotting the center of the upper and lower limit angles. Asdescribed above with reference to FIG. 5, the angle θ must be changedfrom 0.43° to −9.57° depending on the position on the display screen inorder that different images are separately distributed to the left eyeand the right eye. As seen from the graph of FIG. 14C, that conditioncan be satisfied by linearly changing the distance Δy from about 0 to 35μm. Such a linear calculation process is approximately expressed by thefollowing formula (4): $\begin{matrix}{{\theta = {{{- \frac{1}{3.5}}{\nabla y}} \pm 5}}{\theta\text{:}\quad{exit}\quad{angle}}{{\nabla y}\text{:}\quad{lateral}\quad{relative}\quad{position}\quad{of}\quad{light}\quad{emitting}\quad{device}\quad{to}\quad{lens}}} & (4)\end{matrix}$

From the optical calculations described above, it is confirmed thatrespective images dedicated for the left eye and the right eye can bedisplay by using the microlens. While the above description is made, byway of example, of optical design in the left-and-right direction,similar design can be performed in the vertical direction as well. Thedevice size has been discussed herein on assumption of a self-luminousdevice such as a light emitting diode. However, when just limiting theopening size causes a fear of a reduction in the light intensity such asoccurred in the case of using a liquid crystal, an organic EL display, aplasma display, etc., the above-described method can also be appliedafter condensing light from a pixel by a lens or the like into a targetsize of emitted light. The use of the microlens can provide, in additionto high efficiency in utilization of light and low power consumption asmentioned above, other various advantages as follows. Because extraneouslight obliquely entering the microlens is less likely to reflect in thedirection toward the eyes, a high contrast and good quality of an imageare obtained. Because an apparent size of each pixel is increased by thelens effect, an apparent interval between pixels is narrowed and pixelsare avoided from appearing discrete visually. Hence, a continuous imagecan be obtained with a relatively small number of pixels.

One example of the microlenses will be described below with reference toFIGS. 15A and 15B.

FIG. 15A is a front view of the microlenses, and FIG. 15B is a sectionalview of the microlenses. The microlenses shown in FIGS. 15A and 15B hasa structure in which individual microlenses each made of a nearlytransparent spherical body are held at their maximum diameter portionson a holding member substantially in the form of a flat plate, and theyare arrayed substantially at a maximum density. The diameter of eachmicrolens is, e.g., about 300 μm. The image display unit can be formedby bonding, to an array of light emitting devices, the individualmicrolens in a state held on the holding member substantially in theform of a flat plate. Because of no need of positioning the individualmicrolenses one by one, such an assembly process contributes to reducingthe manufacturing cost of the portable communication terminal.

A method of keeping the eye position steady will be described below asone example of image processing that is employed in this embodiment.

Since, as described before with reference to FIG. 1, the cameras 13L,13R for picking up a face image are disposed on both sides of the imagedisplay unit 11 and the user's face is displayed on the minor monitor 14for confirmation, it is possible to make adjustment to some extent suchthat the face is positioned within the image pickup range of thecameras. In the hand-held terminal, however, the positional relationshipbetween the position of the displayed image of the conversation partnerand the cameras greatly vary in usual cases. So long as the face of theconversation partner is substantially included within the displayscreen, a shift of the line-of-sight direction is not extremelyincreased. To achieve more exact eye-to-eye matching and to preventwobbling of the image, however, it is preferable to provide the functionof stabilizing the eye position.

A method of stabilizing the eye position is practiced by preparing amargin in an image area picked up by the cameras 13L, 13R and picking upan image in an area somewhat larger than a face. On the display side,the face image is displayed after adjustment with image processing suchthat the eyes of the conversation partner are positioned on a lineconnecting the cameras 13L, 13R and the center between the eyes islocated closer to the center between those two cameras. Herein, a methodof detecting the eye position from the face image can be practiced byusing a suitable one of well-known image recognition methods. A methodbased on correlation detection will be described below as one example.FIG. 16 is a conceptual view for explaining a process of finding out aportion of an image which is closest to a template. This correlationdetection is performed using a correlation-value calculation formulaexpressed by the following formula (5): $\begin{matrix}\begin{matrix}{{correlation}\quad{coefficient}} & {c_{ij} = \frac{{cov}_{ij}\left( {f,g} \right)}{\sqrt{{var}_{ij}(f)} \times \sqrt{{var}_{ij}(g)}}} \\{covariance} & {{{cov}_{ij}\left( {f,g} \right)} = {\sum\limits_{m = i}^{i + u}{\sum\limits_{n = j}^{j + v}{\left( {f_{m,n} - f^{\prime}} \right)\left( {g_{{m - u},{n - v}} - g^{\prime}} \right)}}}} \\{variance} & {{{var}_{ij}(f)} = {\sum\limits_{m = i}^{i + u}{\sum\limits_{n = j}^{j + v}\left( {f_{m,n} - f^{\prime}} \right)^{2}}}} \\{variance} & {{{var}_{ij}(g)} = {\sum\limits_{m = 0}^{u}{\sum\limits_{n = 0}^{v}\left( {g_{m,n} - g^{\prime}} \right)^{2}}}}\end{matrix} & (5)\end{matrix}$

A matching position is given by coordinate values (i, j) at which acorrelation coefficient c_(ij) in the formula (5) is maximized. Also, inthe formula (5), g represents a template image. In this case, standardimages of eyes, a nose, eyebrows, a mouth, etc. are registered in amemory beforehand. Further, f represents a target image to be displayed.

A generally known method for autofocusing or auto-tracking is alsoapplicable. The cameras 13L, 13R are disposed so as to direct and focusto a standard distance (e.g., L₁=250 mm). Accordingly, the distance to aface is determined from a difference between images picked up by theleft and right cameras. This determination is based on a well-knowngeometrical calculation regarding a parallax. FIG. 17 shows parametersfor a parallax. The calculation is executed using the following formula(6): $\begin{matrix}{{{{{absolute}\quad{parallax}} = {{vergence}\quad{angle}\text{:}\quad\gamma_{F}}},\gamma_{P}}{{parallax} = {{{relative}\quad{parallax}} = {{difference}\quad{between}\quad{vergence}\quad{angles}\text{:}}}}\begin{matrix}{d = {{\gamma_{P} - \gamma_{F}} \approx {E\quad{\delta/\left( {D_{F}^{2} - {\delta\quad D_{F}}} \right)}}}} \\{\approx {E\quad{\delta/D_{F}^{2}}}}\end{matrix}{{{where}\quad D_{F}},{D_{P}\operatorname{>>}E}}} & (6)\end{matrix}$

Based on the distance calculated using the cameras 13L, 13R and thecalculation formula, a standard apparent size of the face is assumed anda template size is adjusted to be matched with the apparent size. Thissize change can be performed by a simple matrix calculation. Such amatrix calculation is expressed, for example, by the following formula(7): $\begin{matrix}\begin{matrix}\begin{matrix}\left. {{normalization}\quad{matrix}}\quad \middle| {{scale}\text{-}{{up}/{down}}\quad{matrix}} \right. \\{{MTRX} = {\begin{bmatrix}{1/X_{s}} & 0 & 0 & 0 \\0 & {{- 1}/X_{s}} & 0 & 0 \\0 & 0 & 1 & 0 \\0 & 0 & 0 & 1\end{bmatrix} \cdot \begin{bmatrix}a & 0 & 0 & 0 \\0 & b & 0 & 0 \\0 & 0 & 1 & 0 \\0 & 0 & 0 & 1\end{bmatrix} \cdot}}\end{matrix} \\\begin{matrix}{{movement}\quad{matrix}} & {z\text{-}{axis}\quad{rotation}\quad{matrix}} \\{\begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & 1 & 0 \\X_{o} & Y_{o} & Z_{o} & 1\end{bmatrix} \cdot} & {\begin{bmatrix}{{Cos}\lbrack w\rbrack} & {{Sin}\lbrack w\rbrack} & 0 & 0 \\{- {{Sin}\lbrack w\rbrack}} & {{Cos}\lbrack w\rbrack} & 0 & 0 \\0 & 0 & 1 & 0 \\0 & 0 & 0 & 1\end{bmatrix} \cdot}\end{matrix} \\\begin{matrix}{x\text{-}{axis}\quad{rotation}\quad{matrix}} & {y\text{-}{axis}\quad{rotation}\quad{matrix}} \\{\begin{bmatrix}1 & 0 & 0 & 0 \\0 & {{Cos}\lbrack u\rbrack} & {{Sin}\lbrack u\rbrack} & 0 \\0 & {- {{Sin}\lbrack u\rbrack}} & {{Cos}\lbrack u\rbrack} & 0 \\0 & 0 & 0 & 1\end{bmatrix} \cdot} & {\begin{bmatrix}{{Cos}\lbrack v\rbrack} & 0 & {{Sin}\lbrack v\rbrack} & 0 \\0 & 1 & 0 & 0 \\{- {{Sin}\lbrack v\rbrack}} & 0 & {{Cos}\lbrack v\rbrack} & 0 \\0 & 0 & 0 & 1\end{bmatrix} \cdot}\end{matrix} \\\begin{matrix}\begin{matrix}{{viewpoint}\quad{coordinate}} \\{{conversion}\quad{matrix}}\end{matrix} & {{projection}\quad{matrix}} & \begin{matrix}{normalization} \\{{inverse}\quad{matrix}}\end{matrix} \\{\begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & {- 1} & 0 \\0 & 0 & t & 1\end{bmatrix} \cdot} & {\begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & {1/s} & {1/s} \\0 & 0 & {- 1} & 0\end{bmatrix} \cdot} & \begin{bmatrix}X_{s} & 0 & 0 & 0 \\0 & {- X_{s}} & 0 & 0 \\0 & 0 & 1 & 0 \\0 & 0 & 0 & 1\end{bmatrix}\end{matrix}\end{matrix} & (7)\end{matrix}$

Instead of the calculations described above, the size change also beperformed by registering a multistage of different sizes in a memorybeforehand and employing the registered data as a lookup table.

After preparing a template for the face with a proper size by any of theabove-described methods, an input image is searched by using thetemplate, and the eye position in the face is determined by finding outa position where the correlation value is maximized. Then, the image isdisplaced with translation, rotation or scale-up/down so that the eyeposition is most exactly superimposed between the left and right imagesand the center between the eyes is positioned close to the centerbetween the two cameras. At this time, the image can be properlydisplayed by using matrices for image conversion, such as expressed bythe above formula (7).

Another example of the structure of the image display unit will bedescribed below with reference to FIGS. 18A and 18B.

FIGS. 18A and 18B show an example of the image display unit employing acone-shaped lens, in which FIG. 18A is a sectional view of the imagedisplay unit, and FIG. 18B is a perspective view of the lens. In theimage display unit, light emitting devices 74, 75, such as lightemitting diodes, are disposed on a support substrate 77. The lightemitting device 74 is arranged substantially at the center of a vacantspace 76 formed by a molded holding member 73 so that an emitted lightbeam is caused to exit nearly in the z-direction. The light emittingdevice 75 is also arranged in the vacant space 76 formed by the moldedholding member 73, but it is disposed at a position shifted in the(−y)-direction substantially from the center of the vacant portion 76 sothat an emitted light beam is caused to exit in a direction shiftedtoward the y-direction. The molded holding member 73 is a memberobtained by molding a suitable synthetic resin. It serves not only as amember for holding the cone-shaped microlenses 71, 72, but also as ashield plate for limiting the direction of light from the light emittingdevice to a proper angle.

In the image display unit shown in FIGS. 18A and 18B, the microlenses71, 72 each have a cone shape and are arranged such that theirbottom-side ends each having a large diameter are positioned to facerespectively the light emitting devices 74, 75 via the vacant space 76and their tapered end surfaces 78, 79 are positioned on the displayscreen side. Particularly, in the microlenses 71, 72, the axialdirections are aligned with the light exit directions. By tilting themicrolenses 71, 72 to direct in respective directions in which lightbeams are to be output, therefore, the light exit angles can be adjustedto desired values and the respective light beams from the light emittingdevices 74, 75 can be condensed and then output from the end surfaces78, 79.

The use of the image display unit having such a structure enables themicrolenses 71, 72, the molded holding member 73, etc. to bemanufactured by the plastic molding technology. Accordingly, productsemploying the image display units can be manufactured at a reduced costwith mass production.

Usefulness resulting from outputting the emitted light beam at an anglenarrowed to a value within 10° will be described below with reference toFIGS. 19 and 20.

FIG. 19 shows a spherical area S corresponding to an included angle φ atthe origin along with related parameters. More specifically, rrepresents the radius of an imaginary sphere, and h represents adiameter of a region where a light beam spreading at the angle φintersects the imaginary sphere. Also, S represents a spherical area cutby a cone with an apical angle φ, and S_(ratio) represents a ratio ofthe spherical area S to a total spherical area. FIG. 20 is a graphshowing the relationship between the angle φ and the spherical arearatio S_(ratio) calculated from formulae shown on the left side of FIG.19. In particular, when the angle φ is 10° (0.1745 rad), the sphericalarea ratio S_(ratio) is a very small value of 0.00190265. In otherwords, comparing with the case of uniformly emitting a light beam over ahemisphere, the amount of light is reduced to 1/263 by emitting a lightbeam at an angle narrowed to within 10°. This suggests that, bycontrolling an angle of the emitted light beam, a good image can beobtained without increasing the amount of light. Also, this means that acontrast can be increased when the light emitting device is driven bythe same electric power, and that a clear eye-to-eye matching face imagecan be displayed during conversation by employing the portablecommunication terminal of this embodiment.

FIG. 21 is a sectional view showing still another example of thestructure of the image display unit. In FIG. 21, light emitting devices83, 84, such as light emitting diodes, are disposed on a substrate 82,and these light emitting devices 83, 84 are arrayed in a matrix pattern.A transparent substrate 81 including small diffraction plates 85L, 85Rformed thereon is bonded to the surface side of the substrate 82. Thesmall diffraction plates 85L, 85R have the functions of bendingrespective light beams emitted from the corresponding light emittingdevices 83, 84 based on the diffraction phenomenon. The light beams bentbased on the diffraction phenomenon are sent toward the user. Morespecifically, the small diffraction plate 85L controls a light exitangle so that the left eye of the user can see the diffracted lightbeam, and the small diffraction plate 85R controls a light exit angle sothat the right eye of the user can see the diffracted light beam. As aresult of such control, the function of looking the different images asa synthesized one works in the cerebrum of the user, and hence the usercan see the images as if an eye-to-eye matching image is displayed onthe screen, as shown in FIG. 2C, whereby communication can be realizedin a natural eye-to-eye matching state.

In the foregoing, the method for constructing the image display unit tohave the mixed pattern of a plurality of pixels for making display inaccordance with the left-eye signal and a plurality of pixels for makingdisplay in accordance with the right-eye signal has been described inconnection with the examples each using the spatially mixed pattern.However, the present invention is not limited to that method, the mixedpattern of a plurality of pixels for making display in accordance withthe left-eye signal and a plurality of pixels for making display inaccordance with the right-eye signal may be prepared by switching overthe display in accordance with the left-eye signal and the display inaccordance with the right-eye signal in a time-division manner.

Results of examining the relationship between the direction of line ofsight and an impression of an eye-to-eye matching image will be brieflydescribed below with reference to FIGS. 22A, 22B, 23A and 23B.

FIGS. 22A and 22B are illustrations showing schemes of experiments toexamine the relationship between the direction of line of sight and animpression of eye-to-eye matching, in which FIG. 22A shows the case inwhich images are picked up by left and right cameras, and FIG. 22B showsthe case in which one camera is assumed to be positioned at the centerof an imaginary screen. The case of FIG. 22A includes a pair of camerasdisposed on both side of the imaginary screen and corresponds to thestructure of the portable communication terminal of this embodiment. Thedistance between the pair of camera is set the same value, i.e., 65 mm,as that between both the eyes. FIGS. 23A and 23B show results of settingobserving points, i.e., 7 points in the vertical direction and 7 pointsin the horizontal direction, on each of the imaginary screens andexamining impressions of images formed under those conditions. Theinterval between two observing points is 12.5 mm that corresponds to anangle of 2.56°. A test participant looks the observing points from aposition 280 mm away from the imaginary screen.

As seen from those results, there is no significant difference betweenthe case in which images are picked up by left and right cameras asshown in FIG. 23A and the case in which one camera is positioned at thecenter of the imaginary screen as shown in FIG. 23B. In other words,picking up images by the left and right cameras can provide theeye-to-eye matching effect comparable to that obtained by positioningone camera at the center of the imaginary screen. Further, the case inwhich images are picked up by the left and right cameras has a strongertendency to give an eye-to-eye matching impression, even when theobserving point is shifted in the vertical direction, than the case inwhich one camera is assumed to be positioned at the center of theimaginary screen. Thus, by employing the portable communication terminalof this embodiment, a satisfactory realism can be realized with theeye-to-eye matching during conversation.

With the portable communication terminal of this embodiment, asdescribed above, by arranging a camera on each of the left and rightsides of the image display unit, the user is able to make conversationwith the conversation partner in an eye-to-eye matching state, and tocontinue the conversation with satisfactory realism. Also, since theefficiency in utilization of light is increased in the portablecommunication terminal, power consumption can be reduced and an imagecan be viewed at a high contrast even in an outdoor bright environment.Further, since the portable communication terminal has the structurethat image pickup devices are disposed on both the left and right sidesof the display screen, the overall size can be reduced and this featureis very useful for applications to portable equipment.

A further improved example of the foregoing portable communicationterminal will be described below.

The portable communication terminal described as the invention isintended to arrange a camera on each of the left and right sides of theimage display unit and to provide stereoscopic display for eye-to-eyematching with the conversation partner based on two images picked up bythe cameras. However, the stereoscopic display for eye-to-eye matchingwith the conversation partner can be optimized and an image being easierto see can be produced by interpolating a parallax based on two imagespicked up by the cameras.

More specifically, when there is a relatively large shift between theimages picked up by two cameras disposed on both the left and rightsides of the image display unit, a parallax is too increased tosynthesize such a double image, as it is, through fusion. Even if thedouble image can be synthesized through fusion, the resulting display isrelatively hard to see and makes the user feel fatigue. In contrast, asshown in FIG. 24, the portable communication terminal described here isintended to, based on two images A_(R), A_(L) picked up by two camerasR_(R), R_(L) disposed on both the left and right sides of the imagedisplay unit, produce new images B_(R), B_(L) having a smaller parallaxas if they are picked up by two imaginary cameras V_(R), V_(L) disposedat a narrower interval than that between the two cameras R_(R), R_(L).Thus, a more natural, much easier-to-see image can be realized bysetting the parallax to a proper value.

While the parallax is generally defined as a relative parallax by adifference between vergence angles as expressed by the formula (6), itis handled herein for the sake of simplicity as the number of pixelsindicating a shift between corresponding points in two images picked upby the two cameras disposed on both the left and right sides of theimage display unit. In the portable communication terminal, the imagerecognition method based on correlation detection can be utilized todetermine corresponding points in two images. More specifically, in theportable communication terminal, as shown in FIG. 25, a group of pixelsin a predetermined area including a face contour with respect to thebackground or a position of an eye or nose, for example, are extractedfrom an image L picked up by the camera disposed on the left side of theimage display unit and are set as a template image g. Then, a targetimage f corresponding to the template image g is searched for from animage R picked up by the camera disposed on the right side of the imagedisplay unit, and the number of pixels indicating a shift between boththe images is determined. In the portable communication terminal, by wayof example, the control circuit 21 described above with reference toFIG. 3, which serves as image processing means, extracts a horizontal2u×vertical 2v region having central coordinate values (i, j) from theimage L and sets it to be the template image g. Then, the controlcircuit 21 searches a horizontal 2u×vertical 2v region having centralcoordinate values (i+p, j+q) with p, q being variable in the image R andfinds out a region where a correlation coefficient c_(ijpq) ismaximized, thereby determining the target image f corresponding to thetemplate image g. At this time, because a target to be picked up isprimarily a face image in the portable communication terminal, a searchrange in the target image f corresponding to the template image g can belimited in advance depending on the position of the template image g inthe image L, and therefore the image processing can be efficientlyexecuted. The correlation detection is performed using acorrelation-value calculation formula expressed by the following formula(8): $\begin{matrix}\begin{matrix}{{correlation}\quad{coefficient}} & {c_{{ij}\quad{pq}} = \frac{{cov}_{{ij}\quad{pq}}\left( {g,f} \right)}{\sqrt{{var}_{ij}(g)} \times \sqrt{{var}_{i + {p\quad j} + q}(f)}}} \\{covariance} & {{{cov}_{{ij}\quad{pq}}\left( {g,f} \right)} = {\sum\limits_{m = {i - u}}^{i + u}{\sum\limits_{n = {j - v}}^{j + v}{\left( {g_{m,n} - g^{\prime}} \right)\left( {f_{{m + p},{n + q}} - f^{\prime}} \right)}}}} \\{variance} & {{{var}_{ij}(g)} = {\sum\limits_{m = {i - u}}^{i + u}{\sum\limits_{n = {j - v}}^{j + v}\left( {g_{m,n} - g^{\prime}} \right)^{2}}}} \\{variance} & {{{var}_{i + {p\quad j} + q}(f)} = {\sum\limits_{m = {i - u}}^{i + u}{\sum\limits_{n = {j - v}}^{j + v}\left( {f_{{m + p},{n + q}} - f^{\prime}} \right)^{2}}}}\end{matrix} & (8)\end{matrix}$

In the formula (8), g′ represents an average value of the templateimage′, and f′ represents an average value of the target image f. Eachof coordinate values (p, q) at which the correlation coefficientc_(ijpq) expressed by the formula (8) is maximized represents the numberof pixels indicating the shift of the target image f relative to thetemplate image g, namely it corresponds to a parallax. In the portablecommunication terminal, therefore, an image having an arbitrary parallaxcan be produced from one of the images R, L by adjusting that shift. Forexample, when an image is produced by moving the image L by the numberof pixels (p/2, q/2) corresponding to ½ of the shift representative ofthe parallax, the portable communication terminal can provide, as shownin FIG. 26, an image picked up from the center between the two cameraspicking up the images R, L, i.e., an image as if it is looked from thefront of an object.

Further, in the portable communication terminal, when a pixel to bedrawn is not present in the original position from which the pixel hasmoved, a vacancy can be filled with a pixel obtained by interpolationfrom pixels positioned, for example, on the lateral side or the lateraland vertical sides of the relevant pixel, whereby a lack can be avoided.Moreover, in the portable communication terminal, when the parallaxbetween the images R and L is so large as to cause the so-calledocclusion, i.e., a concealed portion where some pixels appear in onlyone image, but do not appear in the other image, proper correspondingpoints are often not found. However, because the occlusion also occurswhen a person looks the natural world, the person hardly has an unusualfeeling if a degree of the occlusion is comparable to that in a naturalsituation.

In the portable communication terminal, preferably, the processing todetermine the number of pixels indicating the shift of the target imagef corresponding to the template image g is executed for an entire rangeof the image, and two new images are produced by moving the two imagesR, L by a predetermined number of pixels based on the determined numberof pixels indicating the shift. As a result, the portable communicationterminal can produce two new images having a smaller parallax as if theyare picked up by two imaginary cameras disposed at a narrower intervalthan that between the images R and L. Thus, by displaying those twonewly produced images on the image display unit, the portablecommunication terminal enables the user to see the two images having asmaller parallax corresponding to a shorter distance between the twocameras, and to feel as if a good-quality stereoscopic image isdisplayed which is in an eye-to-eye matching state and is very easy tosee.

Particularly, in the portable communication terminal, by setting theparallax determined from the two images R, L to a value reduced at anydesired rate, a shift between images viewed from the left and right eyesof the user can be reduced and stereoscopic display can be performed inan easier-to-see state.

The parallax interpolation technique can also be used to increase thenumber of viewpoints. Usually, in the portable communication terminal,images looking from two viewpoints can be only produced from the twocameras. Based on those two images, however, images looking from fourviewpoints can be obtained by producing other two images withinterpolation such that the parallax is reduced to, e.g., ½ of theoriginal one. Further, by similarly performing the interpolation so asto reduce the parallax to a predetermined value, the portablecommunication terminal can produce images looking from a larger numberof viewpoints. Then, by properly performing stereoscopic display basedon a plurality of thus-produced images with a reticular lens or thelike, it is possible to reduce the so-called flipping phenomenon thatthe image is abruptly changed depending on a change of the viewpointposition, and to realize better stereoscopic display.

Furthermore, the method of searching the target image f corresponding tothe template image g and determining corresponding points in the twoimages can also be practiced by any other suitable process than thecorrelation detection. For example, under conditions that brightnessvalues of the two images picked up by the two cameras are hardly changedwithout being affected by respective diaphragms, etc. of the twocameras, the so-called Sum-of-Difference method can be used whichutilizes a difference value between the brightness values of the twoimages. The so-called Sum-of-Squared-Difference (SSD) method is alsousable. As a matter of course, however, under conditions that brightnessvalues of the two images picked up by the two cameras differ from eachother, a result with maximum accuracy can be obtained with theabove-described method based on the correlation detection because thosetwo images are normalized.

Additionally, the portable communication terminal executing aninterpolation process for the parallax based on the two images picked upby the cameras disposed on both the left and right sides of the imagedisplay unit may be used on any of the transmitting side and thereceiving side.

With the portable communication terminal of this embodiment, asdescribed above, by interpolating the parallax based on the two imagespicked up by the cameras disposed on both the left and right sides ofthe image display unit, stereoscopic display for eye-to-eye matchingwith the conversation partner can be optimized and a more natural,easier-to-see image can be obtained. Accordingly, it is possible toavoid a situation that the parallax is too increased to synthesize adouble image, as it is, through fusion, and a situation that the imagesynthesized through fusion is so hard to see, thus making the user feelfatigue. Hence, very excellent convenience in use can be provided.

The technique of interpolating the parallax based on the two imagespicked up by the cameras disposed on both the left and right sides ofthe image display unit is applicable to not only the portablecommunication terminal, but also to any other image processing apparatusdisplaying an image.

INDUSTRIAL APPLICABILITY

According to the present invention, as fully described above, the userof the portable communication terminal is able to make conversation inan eye-to-eye matching state with the conversation partner, and tocontinue the conversation with satisfactory realism. Also, since theefficiency in utilization of light is increased, power consumption canbe reduced and an image can be viewed at a high contrast even in anoutdoor bright environment. Further, because of the structure that imagepickup devices are disposed on both the left and right sides of thedisplay screen, the overall size can be reduced and this feature is veryuseful for applications to portable image processing apparatuses.

Moreover, according to the present invention, by producing new imageswith interpolation of a parallax based on two images picked up by imagepickup means, stereoscopic display for eye-to-eye matching with theconversation partner can be optimized and a more natural, easier-to-seeimage can be obtained. Therefore, it is possible to avoid a situationthat the parallax is too increased to synthesize a double image, as itis, through fusion, and a situation that the image synthesized throughfusion is so hard to see, thus making the user feel fatigue. Hence, veryexcellent convenience in use can be provided.

1. An image processing apparatus being portable and used for makingconversation while presenting a picture, said image processing apparatuscomprising: image display means for displaying a desired image inaccordance with an image signal; and image pickup means disposedrespectively on the left and right sides of said image display means. 2.An image processing apparatus according to claim 1, further comprisingimage processing means for producing new images with interpolation of aparallax based on two images picked up by said image pickup means,wherein the two new images produced by said image processing means aredisplayed on a display screen of said image display means.
 3. An imageprocessing apparatus according to claim 2, wherein: said imageprocessing means sets the parallax determined from the two images pickedup by said image pickup means to a value reduced at any desired rate. 4.An image processing apparatus according to claim 2, wherein: said imageprocessing means determines corresponding points in the two imagespicked up by said image pickup means and moves the two images picked upby said image pickup means by a predetermined number of pixels based onthe number of pixels indicating a shift between the determinedcorresponding points, thereby producing the two new images.
 5. An imageprocessing apparatus according to claim 4, wherein: said imageprocessing means determines the corresponding points by detectingcorrelation between the two images picked up by said image pickup means.6. An image processing apparatus according to claim 1, wherein: saidimage pickup means are positioned in zones horizontally aside from leftand right side ends of said image display means having a substantiallyrectangular shape and zones away from upper and lower ends of the leftand right side ends of said image display means within a predetermineddistance.
 7. An image processing apparatus according to claim 1,wherein: said image pickup means are constituted by solid-state imagepickup devices.
 8. An image processing apparatus being portable and usedfor making conversation while presenting a picture, said imageprocessing apparatus comprising: a portable housing; image display meansmounted in a surface of said housing and displaying a desired image inaccordance with an image signal; and image pickup means disposed on thesurface of said housing respectively on the left and right sides of saidimage display means.
 9. An image processing apparatus according to claim8, further comprising image processing means for producing new imageswith interpolation of a parallax based on two images picked up by saidimage pickup means, wherein the two new images produced by said imageprocessing means are displayed on a display screen of said image displaymeans.
 10. An image processing apparatus according to claim 9, wherein:said image processing means sets the parallax determined from the twoimages picked up by said image pickup means to a value reduced at anydesired rate.
 11. An image processing apparatus according to claim 9,wherein: said image processing means determines corresponding points inthe two images picked up by said image pickup means and moves the twoimages picked up by said image pickup means by a predetermined number ofpixels based on the number of pixels indicating a shift between thedetermined corresponding points, thereby producing the two new images.12. An image processing apparatus according to claim 11, wherein: saidimage processing means determines the corresponding points by detectingcorrelation between the two images picked up by said image pickup means.13. An image processing apparatus according to claim 8, wherein: saidhousing has a size allowing a user to hold said housing by one hand, adisplay screen of said image display means has a horizontal width in therange of about 10 mm to about 100 mm, and a distance between said imagepickup means on the left and right sides is set to be larger than thehorizontal width of said display screen.
 14. An image processingapparatus according to claim 13, wherein: said image pickup means arelocated in positions horizontally aside from left and right side ends ofsaid image display means having a substantially rectangular shape and/orpositions away from upper and lower ends of the left and right side endsof said image display means within a predetermined distance.
 15. Animage processing apparatus according to claim 14, wherein: saidpositions away from the upper and lower ends within a predetermineddistance contain positions within a radius of about 20 mm.
 16. An imageprocessing apparatus being portable and used for making conversationwhile presenting a picture, said image processing apparatus comprising:image display means including a mixed pattern of a plurality of pixelsfor making display in accordance with a left-eye signal and a pluralityof pixels for making display in accordance with a right-eye signal; andimage pickup means disposed respectively on the left and right sides ofsaid image display means.
 17. An image processing apparatus according toclaim 16, further comprising image processing means for producing newimages with interpolation of a parallax based on two images picked up bysaid image pickup means, wherein the two new images produced by saidimage processing means are displayed on a display screen of said imagedisplay means.
 18. An image processing apparatus according to claim 17,wherein: said image processing means sets the parallax determined fromthe two images picked up by said image pickup means to a value reducedat any desired rate.
 19. An image processing apparatus according toclaim 17, wherein: said image processing means determines correspondingpoints in the two images picked up by said image pickup means and movesthe two images picked up by said image pickup means by a predeterminednumber of pixels based on the number of pixels indicating a shiftbetween the determined corresponding points, thereby producing the twonew images.
 20. An image processing apparatus according to claim 19,wherein: said image processing means determines the corresponding pointsby detecting correlation between the two images picked up by said imagepickup means.
 21. An image processing apparatus according to claim 16,wherein: said image display means includes output means for outputtinglights to both eyes of a user independently of each other.
 22. An imageprocessing apparatus according to claim 21, wherein said output meanscomprises: light emitting means for emitting desired light in accordancewith the left-eye signal or the right-eye signal; and exit angle controlmeans for controlling the light from said light emitting means to exitin the direction of a predetermined angle.
 23. An image processingapparatus according to claim 22, wherein: said light emitting meanscomprises a plurality of light emitting devices in an array, and saidexit angle control means comprises openings each formed per said lightemitting device and directed toward one of both the eyes of the user.24. An image processing apparatus according to claim 22, wherein: saidexit angle control means is a microlens array comprising a plurality ofsmall lenses arrayed in a matrix pattern, and said light emittingdevices are arranged to be relatively shifted from positions of thecorresponding small lenses in the in-plane direction of said microlensarray such that the lights are output to both the eyes of the userindependently of each other.
 25. An image processing apparatus accordingto claim 24, wherein: a size ratio of said small lens to said lightemitting device is in the range of 30:1 to 5:1.
 26. An image processingapparatus according to claim 24, wherein: said small lens has aspherical, conical, pyramidal, or rectangular shape.
 27. An imageprocessing apparatus according to claim 22, wherein: said light emittingmeans comprises a plurality of light emitting devices in an array, andsaid exit angle control means comprises small diffraction plates arrayedin a matrix pattern and each outputting diffracted light directed towardone of both the eyes of the user.
 28. An image processing apparatusaccording to claim 22, wherein: said exit angle control means provideslinear interpolation of an exit angle of each light emitting meansbetween one end of said image display means on the same side as one eyein the horizontal direction and the other end on the opposite side. 29.An image processing apparatus according to claim 16, wherein: the mixedpattern of a plurality of pixels for making display in accordance with aleft-eye signal and a plurality of pixels for making display inaccordance with a right-eye signal is obtained in said image displaymeans by switching over the display in accordance with the left-eyesignal and the display in accordance with the right-eye signal in atime-division manner.
 30. An information processing system comprising: aplurality of portable information processing terminals used for makingconversation while presenting a picture, each of said informationprocessing terminals comprising image display means capable ofdisplaying an image including a face of a conversation partner, andimage pickup means disposed respectively on the left and right sides ofsaid image display means, said information processing terminals beingable to perform communication therebetween.
 31. An informationprocessing system according to claim 30, wherein: the face of theconversation partner is displayed on a display screen of said imagedisplay means in eye-to-eye matching with a user.
 32. An informationprocessing system according to claim 30, wherein: said informationprocessing terminal includes image processing means for holding an eyeposition on the display screen substantially fixed on the displayscreen.
 33. An information processing system according to claim 31,wherein: each of said information processing terminals further comprisesimage processing means for producing new images with interpolation of aparallax based on two images picked up by said image pickup means, andthe two new images produced by said image processing means are displayedon the display screen of said image display means in said informationprocessing terminal belonging to the conversation partner.
 34. Aconversation partner display method comprising: an image taking-in stepof taking in images of a user by a pair of image pickup means disposedrespectively on the left and right sides of said image display means ina portable terminal; and a display step of displaying the taken-inimages on image display means of a terminal belonging to theconversation partner in eye-to-eye matching between the user and theconversation partner.
 35. A conversation partner display methodaccording to claim 34, further comprising an image processing step ofproducing new images with interpolation of a parallax based on twoimages taken in by said image taking-in step, wherein the two new imagesproduced in said image processing step are displayed on the displayscreen of said image display means in the terminal belonging to theconversation partner.
 36. A conversation partner display methodaccording to claim 34, wherein: said image display means includes amixed pattern of a plurality of pixels for making display in accordancewith a left-eye signal and a plurality of pixels for making display inaccordance with a right-eye signal.
 37. A conversation partner displaymethod according to claim 34, wherein: said image display means includesoutput means for outputting lights to both eyes of the userindependently of each other.
 38. A conversation partner display methodaccording to claim 37, wherein said output means comprises: lightemitting means for emitting desired light in accordance with theleft-eye signal or the right-eye signal; and exit angle control meansfor controlling the light from said light emitting means to exit in thedirection of a predetermined angle.
 39. A conversation partner displaymethod according to claim 34, said terminal comprising: a portablehousing; image display means mounted in a surface of said housing anddisplaying a desired image in accordance with an image signal; and imagepickup means disposed on the surface of said housing respectively on theleft and right sides of said image display means.
 40. An imageprocessing apparatus used for making conversation while presenting apicture, said image processing apparatus comprising: image display meansfor displaying a desired image in accordance with an image signal; imagepickup means disposed respectively on the left and right sides of saidimage display means; and image processing means for producing new imageswith interpolation of a parallax based on two images picked up by saidimage pickup means, wherein the two new images produced by said imageprocessing means are displayed on a display screen of said image displaymeans.
 41. An image processing apparatus used for making conversationwhile presenting a picture, said image processing apparatus comprising:a housing; image display means mounted in a surface of said housing anddisplaying a desired image in accordance with an image signal; imagepickup means disposed on the surface of said housing respectively on theleft and right sides of said image display means; and image processingmeans for producing new images with interpolation of a parallax based ontwo images picked up by said image pickup means, wherein the two newimages produced by said image processing means are displayed on adisplay screen of said image display means.
 42. An image processingapparatus used for making conversation while presenting a picture, saidimage processing apparatus comprising: image display means including amixed pattern of a plurality of pixels for making display in accordancewith a left-eye signal and a plurality of pixels for making display inaccordance with a right-eye signal; image pickup means disposedrespectively on the left and right sides of said image display means;and image processing means for producing new images with interpolationof a parallax based on two images picked up by said image pickup means,wherein the two new images produced by said image processing means aredisplayed on a display screen of said image display means.
 43. Aninformation processing system comprising: a plurality of portableinformation processing terminals used for making conversation whilepresenting a picture, each of said information processing terminalscomprising image display means capable of displaying an image includinga face of a conversation partner, and image pickup means disposedrespectively on the left and right sides of said image display means,wherein each of said information processing terminals further comprisesimage processing means for producing new images with interpolation of aparallax based on two images picked up by said image pickup means, andthe two new images produced by said image processing means are displayedon the display screen of said image display means in said informationprocessing terminal belonging to the conversation partner whencommunication is performed between said information processingterminals.
 44. A conversation partner display method comprising: animage taking-in step of taking in images of a user by a pair of imagepickup means disposed respectively on the left and right sides of imagedisplay means in a terminal; an image processing step of producing newimages with interpolation of a parallax based on two images taken in bysaid image taking-in step; and a display step of displaying the two newimages produced in said image processing step on image display means ofa terminal belonging to a conversation partner such that eye-to-eyematching in display is held between the user and the conversationpartner.